Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Nemo_bis Date: Nov 7, 2013 1:12pm
Forum: web Subject: Re: robots.txt

Hello bmuramatsu, did you manage to whitelist the wayback machine with the Allow directive, in the end? What's your website?

Edit: see also an example of robots.txt which is not respected as intended, Google's. https://archive.org/post/1004436

This post was modified by Nemo_bis on 2013-11-07 21:12:19